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Abstract 

Sparse additive models are families of d-variate functions that have the additive decompo- 
sition /* = X)jes/j*! where S is an unknown subset of cardinality s -C d. In this paper, we 
consider the case where each univariate component function /* lies in a reproducing kernel 
Hilbcrt space (RKHS), and analyze a method for estimating the unknown function /* based on 
kernels combined with ^i-type convex regularization. Working within a high-dimensional frame- 
work that allows both the dimension d and sparsity s to increase with n, we derive convergence 
rates (upper bounds) in the L 2 (P) and L 2 (P n ) norms over the class T^sfli of sparse additive 
models with each univariate function /* in the unit ball of a univariate RKHS with bounded 
kernel function. Wc complement our upper bounds by deriving minimax lower bounds on the 
L 2 (¥) error, thereby showing the optimality of our method. Thus, we obtain optimal minimax 
rates for many interesting classes of sparse additive models, including polynomials, splines, and 
Sobolev classes. We also show that if, in contrast to our univariate conditions, the multivariate 
function class is assumed to be globally bounded, then much faster estimation rates are possible 
for any sparsity s = £l(y/n), showing that global boundedness is a significant restriction in the 
high-dimensional setting. 

1 Introduction 

The past decade has witnessed a flurry of research on sparsity constraints in statistical models. 
Sparsity is an attractive assumption for both practical and theoretical reasons: it leads to more 
interpretable models, reduces computational cost, and allows for model identifiability even under 
high-dimensional scaling, where the dimension d exceeds the sample size n. While a large body 
of work has focused on sparse linear models, many applications call for the additional flexibility 
provided by non-parametric models. In the general setting, a non-parametric regression model 
takes the form y = f*(x\, . . . ,xj) + w, where /* : M. d — > R is the unknown regression function, 
and w is scalar observation noise. Unfortunately, this general non-parametric model is known 
to suffer severely from the so-called "curse of dimensionality", in that for most natural function 
classes (e.g., twice differentiable functions), the sample size n required to achieve any given error 
grows exponentially in the dimension d. Given this curse of dimensionality, it is essential to further 
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constrain the complexity of possible functions /*. One attractive candidate is the class of additive 
non-parametric models [17], in which the function /* has an additive decomposition of the form 

d 

f*(xi,x 2 ,...,x d ) = ^2f*(Xj), (1) 

J'=l 

where each component function /? is univariate. Given this additive form, this function class no 
longer suffers from the exponential explosion in sample size of the general non-parametric model. 
Nonetheless, one still requires a sample size n> d for consistent estimation; note that this is true 
even for the linear model, which is a special case of equation (1). 

A natural extension of sparse linear models is the class of sparse additive models, in which the 
unknown regression function is assumed to have a decomposition of the form 

r( X1 ,x2...,x d ) = j2fj(xj), (2) 

where S C {1,2, ... ,d} is some unknown subset of cardinality |*S r | = s. Of primary interest is the 
case when the decomposition is genuinely sparse, so that s <C d. To the best of our knowledge, 
this model class was first introduced in Lin and Zhang [22], and has since been studied by various 
researchers (e.g., [20, 24, 31, 40]). Note that the sparse additive model (2) is a natural generalization 
of the sparse linear model, to which it reduces when each univariate function is constrained to be 
linear. 

In past work, several groups have proposed computationally efficient methods for estimating 
sparse additive models (2). Just as ^i-based relaxations such as the Lasso have desirable properties 
for sparse parametric models, more general £i-based approaches have proven to be successful in this 
setting. Lin and Zhang [22] proposed the COSSO method, which extends the Lasso to cases where 
the component functions /? lie in a reproducing kernel Hilbert space (RKHS); see also Yuan [40] 
for a similar extension of the non-negative garrote [8]. Bach [3] analyzes a closely related method 
for the RKHS setting, in which least-squares loss is penalized by an £i-sum of Hilbert norms, and 
establishes consistency results in the classical (fixed d) setting. Other related £i-based methods 
have been proposed in independent work by Koltchinskii and Yuan [19] , Ravikumar et al. [31] and 
Meier et al. [24], and analyzed under high-dimensional scaling in <C d). As we describe in more 
detail in Section 3.4, each of the above papers establish consistency and convergence rates for the 
prediction error under certain conditions on the covariates as well as the sparsity s and dimension 
d. However, it is not clear whether the rates obtained in these papers are sharp for the given 
methods, nor whether the rates are minimax-optimal. Past work by Koltchinskii and Yuan [20] 
establishes rates for sparse additive models with an additional global boundedness condition, but 
as will be discussed at more length in the sequel, these rates are not minimax optimal in general. 

This paper makes three main contributions to this line of research. Our first contribution is to 
analyze a simple polynomial-time method for estimating sparse additive models and provide upper 
bounds on the error in the L 2 (P) and L 2 (P n ) norms. The estimator we analyze is based on a 
combination of least-squares loss with two £i-based sparsity penalty terms, one corresponding to 
an £i/L 2 (¥ n ) norm and the other an £i/\\ ■ \\u norm. Our first main result (Theorem 1) shows 

1 The estimator is the same as the estimator analyzed in Koltchinskii and Yuan [20]. We proposed the estimator 
concurrently (see our earlier preprint [29]) and as we discuss later analyze the same estimator under less restrictive 
conditions than those imposed in Koltchinskii and Yuan [20]. 
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that with high probability, if we assume the univariate functions are bounded and independent, 
the error of our procedure in the squared L 2 (P n ) and L 2 (P) norms is bounded by + si/2), 

where the quantity i/ 2 corresponds to the optimal rate for estimating a single univariate function. 
Importantly, our analysis does not require a global boundedness condition on the class J-d,s,"H 
of all s-sparse models, an assumption that is often imposed in classical non-parametric analysis. 
Indeed, as we discuss below, when such a condition is imposed, then significantly faster rates of 
estimation are possible. The proof of Theorem 1 involves a combination of techniques for analyzing 
M-estimators with decomposable regularizers [27] and techniques in empirical process theory for 
analyzing kernel classes [4, 25, 35]. 

Our second contribution is complementary in nature, in that it establishes algorithm-independent 
minimax lower bounds on L 2 (P) error. These minimax lower bounds, stated in Theorem 2, are 
specified in terms of the metric entropy of the underlying univariate function classes. For both 
finite-rank kernel classes and Sobolev-type classes, these lower bounds match our achievable re- 
sults, as stated in Corollaries 1 and 2, up to constant factors in the regime of sub- linear sparsity 
(s = o(d)). Thus, for these function classes, we have a sharp characterization of the associated 
minimax rates. The lower bounds derived in this paper initially appeared in the Proceedings of the 
NIPS Conference (December 2009). The proofs of Theorem 2 is based on characterizing the pack- 
ing entropies of the class of sparse additive models, combined with classical information theoretic 
techniques involving Fano's inequality and variants (see, e.g. the papers [16, 38, 39]). 

Our third contribution is to determine upper bounds on minimax L 2 (P) and L 2 (P n ) error 
when we impose a global boundedness assumption on the class T^sflii meaning that the quantity 
B^d^ft) = supj 6 jr d a H sup x | Ylj=i fj( x j)\ 1S assumed to be bounded independently of (s, d). As 
mentioned earlier, our upper bound in Theorem 1 does not impose a global boundedness condition, 
whereas in contrast, past work by Koltchinskii and Yuan [20] did impose such a global boundedness 
condition in their analysis of the same ^i-kernel-based estimator. Under global boundedness, their 
work provides rates on the L 2 (P) and L 2 (P n ) norm that are of the same order as the results 
presented here. It is natural to wonder whether or not this difference is actually significant — that 
is, do the minimax rates for the class of sparse additive models depend on whether or not global 
boundedness is imposed? In Section 3.5, we define the class J-^ S ^{B) of sparse additive models 
with the additional assumption that B(Fd, s ,H) — B. Theorem 3 and Corollary 3 in this paper 
provide upper bounds on the minimax rate in L 2 (P) and L 2 (P n ) error over the class 7] J? ,(5). 
These rates are faster than those of Theorem 3 in the paper [20] (KY) , showing that the KY rates 
are not minimax optimal for problems with s = £l(y/n). In this way, we see that the assumption 
of global boundedness, though relatively innocuous for classical (low-dimensional) non-parametric 
problems, can be quite limiting in high dimensions. 

The remainder of the paper is organized as follows. In Section 2, we provide background 
on kernel spaces and the class of sparse additive models considered in this paper. Section 3 is 
devoted to the statement of our main results and discussion of their consequences; it includes 
description of our method, the upper bounds on the convergence rate that it achieves, and a 
matching set of minimax lower bounds. Section 3.5 emphasizes the restrictiveness of the global 
uniform boundedness assumption and in particular Theorem 3 and Corollary 3 show that there are 
classes of Sobolev spaces where under the scaling s = Vl{y/n), optimal rates of convergence are faster 
than rates proven in Theorem 2. Section 4 is devoted to the proofs of our three main theorems, with 
the more technical details deferred to the Appendices. We conclude with a discussion in Section 5. 
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2 Background and problem set-up 



We begin with some background on reproducing kernel Hilbert spaces, before providing a precise 
definition of the class of sparse additive models studied in this paper. 

2.1 Reproducing kernel Hilbert spaces 

Given a subset X C R and a probability measure Q on X, we consider a Hilbert space T~L C L 2 (Q), 
meaning a family of functions g : X — > M, with ||ff||x,2(Q) < oo, and an associated inner product 
(■, under which T~L is complete. The space % is a reproducing kernel Hilbert space (RKHS) if 
there exists a symmetric function KiA'xA'-) JR + such that: (a) for each x G X, the function 
x) belongs to the Hilbert space H, and (b) we have the reproducing relation f(x) = (f, K(-,x))u 
for all / G %. Any such kernel function must be positive semidefinite; under suitable regularity 
conditions, Mercer's theorem [26] guarantees that the kernel has an eigen-expansion of the form 

oo 

K(x, x) = £ Hk&kirffaix'), (3) 
fe=i 

where fi± > [ii > > • • • > are a non-negative sequence of eigenvalues, and {(fi^'kLi are the 
associated eigenfunctions, taken to be orthonormal in L 2 (Q). The decay rate of these eigenvalues 
will play a crucial role in our analysis, since they ultimately determine the rate v n for the univariate 
RKHS's in our function classes. 

Since the eigenfunctions {4>k}kLi form an orthonormal basis, any function / £ % has an ex- 
pansion of the f(x) = Y^kLi^kix), where a k = (/, ^ fc ) L 2 ((Q) = J x f{x)<j) k (x) dQ(x) are (gen- 
eralized) Fourier coefficients. Associated with any two functions in % — say / = ^ZfeLi a k<Pk an d 
9 = SfcLi bk<t>k — are t wo distinct inner products. The first is the usual inner product in the space 
L 2 {Q) — namely, (/, g)i 2 (Q) '■ = fx f( x )9( x ) dQ(x). By Parseval's theorem, it has an equivalent rep- 
resentation in terms of the expansion coefficients — namely 

oo 
k=l 

The second inner product, denoted (/, g)u, is the one that defines the Hilbert space; it can be 
written in terms of the kernel eigenvalues and generalized Fourier coefficients as 

oo , 

Using this definition, the Hilbert ball of radius 1 for the Hilbert space T~L with eigenvalues \i k and 
eigenfunctions (j) k (-), is: 

oo oo 2 

B„(l) = {feH; /(•) = £ a k U-) I £—<!}■ (4) 

fc=i k=i ^ k 

For more background on reproducing kernel Hilbert spaces, we refer the reader to various standard 
references [2, 14, 32, 33, 37]. 
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2.2 Sparse additive models over RKHS 

For each j = 1, . . . , d, let Hj C L 2 (Q) be a reproducing kernel Hilbert space of univariate functions 
on the domain <VcK. We assume that 

E[fj(x)] = / fj(x)dQ(x) = for all fj G Hj, and for each j = l,2,...,d. 
Jx 

As will be clarified momentarily, our observation model (7) allows for the possibility of a non-zero 
mean /x, so that there is no loss of generality in this assumption. For a given subset S C {1, 2, . . . , d}, 
we define 

n ( S ) : !•/' Wj, and /j G %. (1) V j G S 1 }, (5) 

corresponding to the class of functions / : X d — > K that decompose as sums of univariate functions 
on co-ordinates lying within the set S. Note that T~L{S) is also (a subset of) a reproducing kernel 
Hilbert space, in particular with the norm 

ies 

where || • denotes the norm on the univariate Hilbert space Hj. Finally, for a cardinality 
s G {1, 2, . . . , [d/2\ }, we define the function class 

Fd,s,H-= U ( 6 ) 
Sc{l,2,...,d} 
\S\=s 

To ease notation, we frequently adopt the shorthand T = J-d,s,Hi but the reader should recall that 
J- depends on the choice of Hilbert spaces {7ij}j =1 , and moreover, that we are actually studying a 
sequence of function classes indexed by (d, s). 

Now let P = Q d denote the product measure on the space X d C W 1 . Given an arbitrary /* G J 7 , 
we consider the observation model 

Vi = (J> + f*(,Xi) +w h for i = 1,2,... ,n, (7) 

where {wi}™^ is an i.i.d. sequence of standard normal variates, and {xi}™ =1 is a sequence of design 
points in M. d , sampled in an i.i.d. manner from P. 

Given an estimate /, our goal is to bound the error f — f* under two norms. The first is the 
usual 1? (P) norm on the space given the product structure of P and the additive nature of any 
/ G J 7 , it has the additive decomposition ||/||^ 2 /p) = Ylj=i \\fj\\ 2 L^(Q)- ^ n addition, we consider the 
error in the empirical L 2 (¥ n )-norm defined by the sample {^i}™ =1 , defined as 

1 n 

imiW=-£/ 2 (^)- 

i=l 

Unlike the L 2 (P) norm, this norm does not decouple across the dimensions, but part of our analysis 
will establish an approximate form of such decoupling. For shorthand, we frequently use the 
notation ||/||2 = ||/||l 2 (P) an d ||/||n = ||/||l 2 (f„) f° r a d-variate function / G T. With a minor 
abuse of notation, for a univariate function fj G Hj, we also use the shorthands ||/j||2 = ||/illL 2 (Q) 
and \\fj\\ n = ||/j|| L 2 (Qn) . 
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3 Main results and their consequences 



This section is devoted to the statement of our three main results, and discussion of some of 
their consequences. We begin in Section 3.1 by describing a regularized M-estimator for sparse 
additive models, and we state our upper bounds for this estimator in Section 3.2. This estimator 
is essentially equivalent to that analyzed in the paper KY [20], except that we allow for a non-zero 
mean for the function, and estimate it as well. We illustrate our upper bounds for various concrete 
instances of kernel classes. In Section 3.3, we state minimax lower bounds on the L 2 (P) error over 
the class Td,sflii which establish the optimality of our procedure. In Section 3.4, we provide a 
detailed comparison between our results to past work, and in Section 3.5 we discuss the effect of 
global boundedness conditions on optimal rates. 



3.1 A regularized M-estimator for sparse additive models 

For any function of the form / = $Zj=i fj, the (L 2 (Q n ), 1) and (T~L, l)-norms are given by 

d d 

ll/lki: = Ell/illn, and ll/H«,i: = Ell/ill«> ^ 

3=1 3=1 

respectively. Using this notation and defining the sample mean y n = \ Y^i=i Vii we define the cost 
functional 

1 n 

C ^ = T E (V* - ~ + X n\\f\\n,l + Pnll/lkl- (9) 

i=l 

The cost functional C{f) is least-squares loss with a sparsity penalty an d a smoothness 

penalty H/11^,1. Here (X n ,Pn) are a pair of positive regularization parameters whose choice will be 
specified by our theory. Given this cost functional, we then consider the M-estimator 

/ £ argmin£(/) subject to / = Y$=i fj and WfjWn < 1 for a11 3 = I, 2 , ■ ■ ■ ,d. (10) 

/ 

In this formulation (10), the problem is infinite-dimensional in nature, since it involves optimization 
over Hilbert spaces. However, an attractive feature of this M-estimator is that, as a straightforward 
consequence of the representer theorem [18, 33], it can be reduced to an equivalent convex program 
in W 1 x R d . In particular, for each j = 1, 2, . . . , d, let KP denote the kernel function for co-ordinate 
j. Using the notation X{ = (xn,Xi2, ■ ■ ■ , Xid) for the i th sample, we define the collection of empirical 
kernel matrices K 3 S M nxn , with entries = M. J (xij,xij). By the representer theorem, any 
solution / to the variational problem (10) can be written in the form 

n d 

f{zi, ...,z d ) = ^^JayK J (^,ijj), 
i=i j=i 

for a collection of weig hts {aij £ W 1 , j = 1, . . . ,d}. The optimal weights are obtained by solving 
the convex program 



fai, . . . , atd) G arg min 

cgKiaj<l 

J J 



s - d d " d >. 

\^-\\y-yn-J2 K3a ^ + A « E v - W Kja i Hi + Pn E \M^3 • 

L .7=1 .7 = 1 .7=1 J 



(11) 
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This problem is a second-order cone program (SOCP), and there are various algorithms for finding 
a solution to arbitrary accuracy in time polynomial in (n,d), among them interior point methods 
(e.g., see §11 in the book [7]). 

Various combinations of sparsity and smoothness penalties have been used in past work on sparse 
additive models. For instance, the method of Ravikumar et. al [31] is based on least-squares loss 
regularized with single sparsity constraint, and separate smoothness constraints for each univariate 
function. They solve the resulting optimization problem using a back-fitting procedure. Koltchinskii 
and Yuan [19] develop a method based on least-squares loss combined with a single penalty term 
X)j=i ll/ill'H- Their method also leads to an SOCP if H is a reproducing kernel Hilbert space, 
but differs from the program (11) in lacking the additional sparsity penalties. Meier et. al [24] 

analyzed least-squares regularized with a penalty term of the form Y^j=i \j ^ill/jlln + ^H/jll^, 
where Ai and A2 are a pair of regularization parameters. In their method, Ai controls the sparsity 
while A2 controls the smoothness. If H is an RKHS, the method in Meier et. al [24] reduces to 
an ordinary group Lasso problem on a different set of variables, which can be cast as a quadratic 
program. The more recent work of Koltchinskii and Yuan [20] is based on essentially the same 
estimator as problem (10), but they impose stronger assumptions in their analysis. We provide a 
more in-depth comparison of our analysis and results with the past work listed above in Sections 3.4 
and 3.5. 



3.2 Upper bound 

We now state a result that provides upper bounds on the estimation error achieved by the estima- 
tor (10), or equivalently (11). To simplify presentation, we state our result in the special case that 
the univariate Hilbert space "Hj, j = 1, . . . ,d are all identical, denoted by T~L. However, the analysis 
and results extend in a straightforward manner to the general setting of distinct univariate Hilbert 
spaces, as we discuss following the statement of Theorem 1. 

Let fi± > fj,2 > • • • > denote the non-negative eigenvalues of the kernel operator defining the 
univariate Hilbert space 7i, as defined in equation (3), and define the function 

1 00 

Q ff ,„(t):=^[J>in{t 2 ,M] 1/2 - (12) 

Let v n > be the smallest positive solution to the inequality 

40^ > Q^ n (v n ), (13) 

where the 40 is simply used for technical convenience. We refer to v n as the critical univariate rate, 
as it is the minimax-optimal rate for I? (P)-estimation of a single univariate function in the Hilbert 
space H. (e.g., [25, 35]). This quantity will be referred to throughout the remainder of the paper. 
Our choices of regularization parameters are specified in terms of the quantity 



7„ := Kmax{f n ,y^^}, (14) 

where k is a fixed constant that we choose later. We assume that each function within the unit 
ball of the univariate Hilbert space is uniformly bounded by a constant multiple of its Hilbert 
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norm — that is, for each j = 1, . . . , d and each /,• S T~L, 

ll/jlloo : = sup \fj{xj)\ <c||/j||^. (15) 

This condition is satisfied for many kernel classes including Sobolev spaces, and any univariate 
kernel function 2 bounded uniformly by c. Such a condition is routinely imposed for proving upper 
bounds on rates of convergence for non-parametric least squares in the univariate case d = 1 (see 
e.g. [34, 35]). Note that this univariate boundedness does not imply that the multivariate functions 
/ = X^/es fj i n 3~ are uniformly bounded independently of (d, s); rather, they can take on values 
of the order y^s. 

The following result applies to any class J-^s/h of sparse additive models based on the univariate 
Hilbert space satisfying condition (15), and to the estimator (10) based on n i.i.d. samples (xi, 2/i)?=i 
from the observation model (7). 

Theorem 1. Let f be any minimizer of the convex program (10) with regularization parameters 
A n > 167n and p n > I67 2 . Then provided that wy^ = f2(log(l/7 n )), there are universal constants 
(C, 01,02) such that 



max{||/ - ml, ||/ - /* II*} > C{s\l + s Pn ] 
We provide the proof of Theorem 1 in Section 4.1. 



< ciexp(-c 2 ?77 2 ). (16) 



Remarks: First, the technical condition = 0(log(l/7 n )) is quite mild, and satisfied in most 
cases of interest, among them the kernels considered below in Corollaries 1 and 2. 

Second, note that setting A n = C7„ and p n = C7 2 for some constant c £ (16, 00) yields the rate 
®( s 7n + s Pn) = 6( sl ° grf + su^). This rate may be interpreted as the sum of a subset selection 
term ( s *° g - ) and an s-dimensional estimation term (sv^). Note that the subset selection term 
( sl °n d ) is independent of the choice of Hilbert space T~L whereas the s-dimensional estimation term 
is independent of the ambient dimension d. Depending on the scaling of the triple (n, d, s) and the 
smoothness of the univariate RKHS rl, either the subset selection term or function estimation term 
may dominate. In general, if = o(z/ 2 ), the s-dimensional estimation term dominates, and vice 
versa otherwise. At the boundary, the scalings of the two terms are equivalent. 

Finally, for clarity, we have stated our result in the case where the univariate Hilbert space T~L 
is identical across all co-ordinates. However, our proof extends with only notational changes to the 
general setting, in which each co-ordinate j is endowed with a (possibly distinct) Hilbert space rlj. 
In this case, the Af-estimator returns a function / such that (with high probability) 

max{||/-/X ||/- /111} < Cj^ + g^j, 



2 Indeed, we have 

sup|//(ay)l = swp\(fj(.),K(.,xj)} H \ < sup s/K(xj,Xj)\\fj\\n. 
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where v n j is the critical univariate rate associated with the Hilbert space T~Lj, and S is the subset 
on which /* is supported. 



Theorem 1 has a number of corollaries, obtained by specifying particular choices of kernels. 
First, we discuss m-rank operators, meaning that the kernel function K can be expanded in terms 
of m eigenfunctions. This class includes linear functions, polynomial functions, as well as any 
function class based on finite dictionary expansions. First we present a corollary for finite-rank 
kernel classes. 

Corollary 1. Under the same conditions as Theorem 1, consider an univariate kernel with finite 
rank m. Then any solution f to the problem (10) with X n = cry n and p n = cy 2 with 16 < c < oo 
satisfies 



max{n/ - rn 2 n) ii/ - rm * c i— + *-} 

rt n 



< c\ exp (- c 2 (m + log d)). (17) 



Proof. It suffices to show that the critical univariate rate (13) satisfies the scaling f 2 = 0(m/n). 
For a finite-rank kernel and any t > 0, we have 



Q*,n(t) 

m 



m I 

^min^ 2 ,^} < 

.7 = 1 



3- 

from which the claim follows by the definition (13). □ 

Next, we present a result for the RKHS's with infinitely many eigenvalues, but whose eigenvalues 
decay at a rate /i^ ~ (l/k) 2a for some parameter a > 1/2. Among other examples, this type of 
scaling covers the case of Sobolev spaces, say consisting of functions with a derivatives (e.g., [6, 15]). 

Corollary 2. Under the same conditions as Theorem 1, consider an univariate kernel with eigen- 
value decay ~ (l/k) 2a for some a > 1/2. Then the kernel estimator defined in (10) with 
An = cj n and p n = cy\ with 16 < c < oo satisfies 



max{||/ - HI ||/ - ml} > + 

rt n' ' 



< ci exp (- c 2 (n 2<*+i + log d)). (18) 



Proof. As in the previous corollary, we need to compute the critical univariate rate v n . Given the 

!_ 1 

assumption of polynomial eigenvalue decay, a truncation argument shows that Q a ,n(t) = O ( 1 ^£ ) . 
Consequently, the critical univariate rate (13) satisfies the scaling u 2 x v n 2a / 'y/n, or equivalently, 



3.3 Minimax lower bounds 

In this section, we provide minimax lower bounds in L 2 (P) error so as to complement the achiev- 
ability results derived in Theorem 1. Given the function class J 7 , the minimax L 2 (P)-error is given 
by 

9%(.Fd, a ,«) : = mf sup ||/ n - /*|||, (19) 
fn /*eJ- 
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where the infimum is taken over all measurable functions of n samples {(xj, Vi)}" = i- As defined, 
this minimax error is a random variable, and our goal is to obtain a lower bound in probability. 

Central to our proof of the lower bounds is the metric entropy structure of the univariate re- 
producing kernel Hilbert spaces. More precisely, our lower bounds depend on the packing entropy, 
defined as follows. Let (G,p) be a totally bounded metric space, consisting of a set Q and a metric 
p : Q x Q — > R + . An e-packing of Q is a collection {/\ . . . , f M } C Q such that p(f\ f j ) > e for 
all i ^ j. The e-packing number M(e;Q,p) is the cardinality of the largest e-packing. The packing 
entropy is the simply the logarithm of the packing number, namely the quantity log M(e; Q, p), to 
which we also refer as the metric entropy. In this paper, we derive explicit minimax lower bounds 
for two different scalings of the univariate metric entropy. 

Logarithmic metric entropy: There exists some m > such that 

logM(e;JB w (l),L 2 (P)) ~ m log(l/e) for all e £ (0,1). (20) 

Function classes with metric entropy of this type include linear functions (for which m = k), 
univariate polynomials of degree k (for which m = k + 1), and more generally, any function space 
with finite VC-dimension [36] . This type of scaling also holds for any RKHS based on a kernel with 
rank m (e.g., see [11]), and these finite-rank kernels include both linear and polynomial functions 
as special cases. 

Polynomial metric entropy There exists some a > such that 

logM(e;B w (l),L 2 (P))~(l/e) 1 / Q for all e G (0, 1). (21) 

Various types of Sobolev/Besov classes exhibit this type of metric entropy decay [6, 15]. In fact, 
any RKHS in which the kernel eigenvalues decay at a rate k~ 2a have a metric entropy with this 
scaling [10, 11]. 

We are now equipped to state our lower bounds on the minimax risk (19): 

Theorem 2. Given n i.i.d. samples from the sparse additive model (7) with sparsity s < d/4, 
there is an universal constant C > such that: 

(a) For a univariate class rl with logarithmic metric entropy (20) indexed by parameter m, we 
have 

m<?w) > c{^M + s -l (22) 

y n n J 

with probability greater than 1/2. 

(b) For a univariate class Ti with polynomial metric entropy (21) indexed by a, we have 

m^H) >C{ S -^IH +S {^\ (23) 
[ n n J 

with probability greater than 1/2. 
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The proof of Theorem 2 is provided in Section 4.2. Our choice of stating bounds that hold with prob- 
ability 1 /2 is simply a convention often used in information-theoretic approaches (see, for instance, 
the papers [16, 38, 39]). We note that analogous lower bounds can established with probabilities 
arbitrarily close to one, albeit at the expense of worse constants. The most important consequence 
of Theorem 2 is in establishing the minimax-optimality of the results given in Corollary 1 and 2; in 
particular, in the regime of sub-linear sparsity (i.e., for which logd = 0(\og(d/s))), the combination 
of Theorem 2 with these corollaries identifies the minimax rates up to constant factors. 

3.4 Comparison with other estimators 

It is interesting to compare these convergence rates in L 2 (P n ) error with those established in past 
work [19, 24, 31] using different estimators. Ravikumar et. al [31] show that any solution to their 
back-fitting method is consistent in terms of mean-squared error risk (see Theorem 3 in their pa- 
per) but they don't have rates in the regime where s — > oo. An earlier method of Koltchinskii 
and Yuan [19] is based regularizing the least-squares loss with the (T~L, l)-norm penalty — that is, 
SjLi ll/jll'H but no (||.|| n , l)-norm penalty; Theorem 2 in their paper presents a rate that cap- 
tures the decomposition into two terms, a subset selection and s-dimensional estimation term. In 
quantitative terms however, their rates are looser than those given here; in particular, their bound 
includes a term of the order s ° s , which is larger than the bound in Theorem 1. Meier et al. [24] 
analyze a different M-estimator to the one we analyze in this paper. Rather than adding two 
separate (%,l)-norm and an (||.||n> l)-norm penalties, they combine the two terms into a single 
sparsity and smoothness penalty. For their estimator, Meier et al. [24] establish a convergence rate 
of the form 0(s(^p)^) in the case of a-smooth Sobolev spaces (see Theorem 1 in their paper) . 
This result is sub-optimal compared to the optimal rate proven in Theorem 2(b). More precisely, 
we either have < (l^Si^) 2 Q +i when subset selection term dominates, or (-) 2a + 1 < (l^ifh 2«+i , 
when the s-dimensional estimation term dominates. In all of the above-mentioned methods, it 
is unclear whether or not sharper analysis would yield better rates. Koltchinskii and Yuan [20] 
analyzes the same estimator as the M-estimator (10) and achieve the same rates as in Theorem 1, 
under a global boundedness condition. In the following section, we show that rates in their paper 
are not minimax optimal for Sobolev spaces when s = £l(^/n). 

3.5 Upper bounds under a global boundedness assumption 

As discussed previously in the introduction, the past work of Koltchinski and Yuan [20], referred 
to as KY for short, is based on the M-estimator (10). In terms of rates obtained, they establish a 
convergence rate based on two terms as in Theorem 1, but with a pre- factor that depends on the 
global quantity 

B= sup H/lloo = sup sup |/(x)|, (24) 

assumed to be bounded independently of dimension and sparsity. Such types of global boundedness 
conditions are fairly standard in classical non-parametric estimation and is equivalent to univariate 
boundedness, and they have no effect on minimax rates. In sharp contrast, the analysis of this 
section shows that for sparse additive models in the regime s = ^L(^/n), such global boundedness 
can substantially speed up minimax rates, showing that the rates proven in KY are not minimax 
optimal for these classes. The underlying insight is as follows: when the sparsity grows, imposing 



11 



global boundedness over s-variate functions substantially reduces the effective dimension from its 
original size s to a lower dimensional quantity, which we denote by sKb(s,u), and moreover, the 
quantity K B {s, n) — > when s = Q(^/n) as described below. 

Recall the definition (6) of the function class F^s H- The model considered in the KY paper is 
the smaller function class 

5c{l,2,...,d} 
\S\=s 

where H(S, B):={f = J2 j£S Sj I Sj G and £ 6 %(1) Vj £ 5 and < 5}. 

The following theorem provides sharper rates for the Sobolev case, in which each univariate 
Hilbert space has eigenvalues decaying as fik — k~ 2a for some smoothness parameter a > 1/2. Our 
probabilistic bounds involve the quantity 



i 



./slog(d/s) D i /2 ,a»loga 
n ' n 



)S i/ 2( £^g £) i/4 )) (25) 



and our rates are stated in terms of the function 

K B (s,n) : = B^i7(s- 1 /2« n V(4a+2) ) 2a-i_ (26) 

Note that K B {s,n) — > if s = fi(-^/n). With this notation, we have the following upper bound on 
the minimax risk over the function class J-^ s %(-£?)• 

Theorem 3. Consider a Sobolev RKHSU with eigenvalue decay k~ 2a and eigenf unctions such that 
ll^fclloo < C < oo. Then there are universal constants (cx,C2, k) such that with probability greater 
than 1 — 2exp ( — c\n8T\, we have 



\\T- f*\\l < k 2 {1 + B)Csn-^TT (^K B {s,n) + ra- 1/{2a+1) log(d/s)^ , (27) 

^(^, s , H (s)) 



min max 



as /ong as n^ 2 = f2(log(l/5 n )). 



We provide the proof of Theorem 3 in Section 4.3; it is based on analyzing directly the least- 
squares estimator over J~d s u(B). The assumption that Halloo < C < oo for all k includes the 
usual Sobolev spaces in which are (rescaled) Fourier basis functions. An immediate consequence 
of Theorem 3 is that the minimax rates over the function class J-^ S ^(B) can be strictly faster than 
minimax rates for the class J-d,s,n (which does not assume global boundedness). Recall that the 
minimax lower bound from Theorem 2 (b) takes the form: 

attpfo*,*) := min max ||/ - /* \\ 2 2 > dsn^ ( 1 + „-V(2«+i) log(d/s) 
/ f*^d.s,n \ 

for a universal constant C\. Note that up to constant factors, the achievable rate (27) from 
Theorem 3 is the same except that the term 1 is replaced by the function Kb(s, n). Consequently, 
for scalings of (s,n) such that KB(s,n) — > 0, global boundedness conditions lead to strictly faster 
rates. 
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Corollary 3. Under the conditions of Theorem 3, we have 



WAFXsd B ^ ~ CK 2 (l+B){K B ( S ,n)+n- l /V<*+i)\og{d/s)) 
with probability at least 1/2, whenever B = 0(1) and Ks(s,n) — > 0. 



Remarks: The quantity i^s(s,n) is guaranteed to decay to zero as long as the sparsity index s 
grows in a non-trivial way with the sample size. For instance, if we have s = 0( v / n) for a problem of 
dimension d = 0{n@) for any (3 > 1/2, then it can be verified that /_b(s, n) = o(l). As an alternative 
view of the differences, it can be noted that there are scalings of (n, s, d) for which the minimax rate 
%Rp(J~d,s,H) ° ver J~d,s,y. is constant — that is, does not vanish as n — > +oo — while the minimax rate 
9Jtp(J r ( J s u(B)) does vanish. As an example, consider the Sobolev class with smoothness a = 2, cor- 
responding to twice-differentiable functions. For a sparsity index s = C(n 4//5 ), then Theorem 2(b) 
implies that ^01^(^,3, n) = ^(1), so that the minimax rate over Fd^fli ls strictly bounded away 
from zero for all sample sizes. In contrast, under a global boundedness condition, Theorem 3 shows 
that the minimax rate is upper bounded as 9Jtp(J r ^ S - H {B)) = 0(n -1 ' 5 y / Iogn) , which tends to zero. 

In summary, Theorem 3 and Theorem 2(b) together show that the minimax rates over Td, s fli 
and J-^ s u(B) can be drastically different. Thus, global boundedness is a stringent condition in the 
high-dimensional setting; in particular, the rates given in Theorem 3 of Koltchinskii and Yuan [20] 
are not minimax optimal when s = f2(\/n). 



4 Proofs 

In this section, we provide the proofs of our three main theorems. For clarity in presentation, we 
split the proofs up into a series of lemmas, with the bulk of the more technical proofs deferred to 
the appendices. This splitting allows our presentation in Section 4 to be relatively streamlined. 

4.1 Proof of Theorem 1 

At a high-level, Theorem 1 is based on an appropriate adaptation to the non-parametric setting 
of various techniques that have been developed for bounding the error to those in sparse linear 
regression (e.g., [5, 27]). In contrast to the parametric setting where classical tail bounds are 
sufficient, controlling the error terms in the non-parametric case requires more advanced techniques 
from empirical process theory. In particular, we make use of concentration theorems for Gaussian 
and empirical processes (e.g., [12, 21, 23, 28, 35]) as well as results on the Rademacher complexity 
of kernel classes [4, 25]. 

At the core of the proof are three technical lemmas. First, Lemma 1 provides an upper bound 
on the Gaussian complexity of any function of the form / = Y^j=i fj m t erms of the norms || • 
and || • ||„i previously defined. Lemma 2 exploits the notion of decomposability [27], as applied 
to these norms, in order to show that the error function belongs to a particular cone-shaped set. 
Finally, Lemma 3 establishes an upper bound on the L 2 (P) error of our estimator in terms of 
the L 2 (P n ) error which completes the proof. The latter lemma can be interpreted as proving 
that our problem satisfies non-parametric analog of a restricted strong convexity [27] or restricted 
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eigenvalue condition [5]. The proof of Lemma 3 involves a new approach that combines the Sudakov 
minoration [28] with a one-sided concentration bound for non-negative random variables [12]. 

Throughout the proof, we use C and q, i = 1, 2, 3, 4 to denote universal constants, independent 
of (n, d, s). Note that the precise numerical values of these constants may change from line to line. 
The reader should recall the definitions of v n and j n from equations (13) and (14) respectively. 
For a subset AC{l,2,...,d} and a function of the form / = X^'=i /i> we adopt the convenient 
notation 

\\f A \\n,i := J2\\fj\\n, and ||/ A || Wjl := £ (28) 

We begin by establishing an inequality on the error function A : = / — /*. Since / and /* are, 
respectively, optimal and feasible for the problem (10), we are guaranteed that C(f) < £(/*), and 
hence that the error function A satisfies the bound 

1 n ^ l n 

— J2(Wi+(J,-y n - A(x i )) 2 +A n ||/|| n ,i+p n ||/|| W ,i < — (m + M - f7n) 2 +An||rikl+Pn||/ltt,l- 
i=l i=l 

Some simple algebra yields the bound 

1 ~ 1 n ~ 1 ™ - 

-\\A\\ 2 n < \-J2 Wi A(xi)\ + \y n -n\\-^2A( Xi )\ + X n \\A\\ njl + p n \\A\\ Hjl . (29) 
i=i i=i 

Following the terminology of van de Geer [35], we refer to this bound as our basic inequality. 



4.1.1 Controlling deviation from the mean 

Our next step is to control the error due to estimating the mean \y n — fi\. We begin by observing 
that this error term can be written as y n — [i = - 5Zi=iG/» — Z 1 )' Now consider the random variable 
yi — [i = ^2j & s fj( x ij) + w i- ^ i s the sum of the s independent random variables fj(xij), each 
bounded in absolute value with one, along with the independent sub-Gaussian noise term wf, 
consequently, the variable yi — \i is sub-Gaussian with parameter at most \J s + 1 (see, e.g., Lemma 
1.4 in Buldygin and Kozachenko [9]). By applying standard sub-Gaussian tail bounds, we have 
P(|y n — fj,\ > t) < 2 exp(— 2 (s+i) )> an d hence, if we define the event C(j n ) = {\y n — fj,\ < y^Tn}, we 
are guaranteed 

P[C( 7re )]>l-2ex P (-^). 

For the remainder of the proof, we condition on the event C(j n ). Under this conditioning, the 
bound (29) simplifies to: 

1 ~ l n 

-\\A\\ 2 n < \~y~]wiA(xi)\ + v^TnUAHn + A n ||A|| n ,l + yO„||A|| Wi i, 
i=l 

where we have applied the Cauchy-Schwarz inequality to write |^ ^1=1 ^( x «)| — II^IU- 
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4.1.2 Controlling the Gaussian complexity term 

The following lemma provides control the Gaussian complexity term on the right-hand side of 
inequality (29) by bounding the Gaussian complexity for the univariate functions Aj, j = 1, 2, . . . , d 

in terms of their || • || n and || • ||% norms. In particular, recalling that j n = ft max { y^j^, v n }, we 
have the following lemma. 

Lemma 1. Define the event 

T(7n):={v j = l,2,...,d, \-J2 w i&j( x ij)\ < 8 7n fe||« + 87„ ||Aj|| n ). (30) 
1 n i=i J 

Then under the condition wy^ = 0(log(l/7 n )) ; we have 

P(T(7n)) > 1 - ci exp(-c 2 n 7 2). (31) 

The proof of this lemma, provided in Appendix B, uses concentration of measure for Lipschitz 
functions over Gaussian random variables [21] combined with peeling and weighting arguments [1, 
35]. In particular, the subset selection term ( s *° g d ) in Theorem 1 arises from taking the maximum 
over all d components. 

The remainder of our analysis involves conditioning on the event 7~( 7 n)nC(7 n ). Using Lemma 1, 
when conditioned on the event T(7n) HC(7 n ) we have: 

||A||2 < 2v^n||A|| n + (167 n + 2A n )||A|| n ,i + (lfr^ + 2p n )||A||«,i. (32) 



4.1.3 Exploiting decomposability 

Recall that S denotes the true support of the unknown function /*. By the definition (28), we 
can write ||A|| nj i = ||A 5 || ni i + ||A 5 c|| n)1 , where A s : = Y^jeS A j and A S c := Z^eS c Ar Similarly, 
we have an analogous representation for ||A||^i. The next lemma shows that conditioned on the 
event 7~(7 n ), the quantities ||A||% i and ||A|| n i are not significantly larger than the corresponding 
norms as applied to the function Ag. 

Lemma 2. Conditioned on the events T("f n ) and C( 7n ), and with the choices X n > 16j n and 
Pn > 167^, we have 

A n ||A|| nj i + /9 n ||A|| Wi i < 4A n ||As|| re ,i + 4p ri ||As|| W) i + -s^. (33) 

The proof of this lemma, provided in Appendix C, is based on the decomposability [27] of the || • 
and || • || n 1 norms. This lemma allows us to exploit the sparsity assumption, since in conjunction 
with Lemma 1, we have now bounded the right-hand side of the inequality (32) by terms involving 
only A s . 

For the remainder of the proof of Theorem 1, we assume A n > 167 n and p n > 167^. In 
particular, still conditioning on C( 7n ) D T(7 n .) and applying Lemma 2 to inequality (32), we obtain 

\\A\\n < 2Vs7 n ||A|| n + 3A n ||A|| nj i + 3p n ||A|| Wi i 

^ ^ ~ 3 

< 2\/sA n ||A|| n + 12A n ||As|| n) i + 12p n \\As\\n,i + —sp n , 
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Finally, since both fj and /? belong to B%(1), we have ||Aj||% < + ll/jll'H ^ 2, which implies 

that HAsH^i < 2s, and hence 

\\A\\ 2 n < 2 v / iA n ||A||„ + 12A n ||A s || n>1 + 25sp n . (34) 
4.1.4 Upper bounding ||As|| n) i 

The final step is to control the term ||As|| ni i = Y^jes IIAjlU that appears in the upper bound (34). 
Ideally, we would like to upper bound it by a quantity of the order y'sll A,s|| 2 . Such an upper bound 
would follow immediately if it were phrased in terms of the population || • ||2-norm rather than the 
empirical- 1 1 • || n norm, but there are additional cross-terms with the empirical norm. Accordingly, 
a somewhat more delicate argument is required, which we provide here. First define the events 

Aj(X n ) : = {||Aj|| n < 2||A,-|| 2 + A n }, 

and A(X n ) = rij =1 ^4j(A n ). By applying Lemma 7 from Appendix A with t = X n > 167„ and 
b = 2, we conclude that \\fj\\ n < 2||/j|| 2 + A n . with probability greater than 1 — c\ exp(— c 2 nA^). 
Consequently, if we define the event A(X n ) = Dj£sAj(X n ), then this tail bound together with the 
union bound implies that 

P[A c (X n )} < s c 1 exp(-c 2 n\l) < ciexp(-c' 2 nX 2 n ), (35) 



where we have used the fact that A n = ^(y "ip)- Now, conditioned on the event ^4(A n ), we have 
||A 5 |U,i < ^IIAjH,, < 2^||A i || 2 + sA„ < 2^||A 5 || 2 + sA n < 2^\\A\\ 2 + sX n . (36) 

Substituting this upper bound (36) on HAsH^i into our earlier inequality (34) yields 

||A||2 < 2^A n ||A|| n + 24^A n ||A|| 2 + 12sA2 +25sp n . (37) 

At this point, we encounter a challenge due to the unbounded nature of our function class. In 
particular, if ||A|| 2 were upper bounded by C max(||A|| n , y/sX n , yjsp n ), then the upper bound (37) 
would immediately imply the claim of Theorem 1 . If one were to assume global boundedness of the 
multivariate functions / and /*, as done in past work [20], then an upper bound on ||A|| 2 of this 
form would directly follow from known results (e.g., Theorem 2.1 in Bartlett et al. [4].) However, 
since we do not impose global boundedness, we need to develop a novel approach to obtaining 
suitable control ||A|| 2 , the task to which we now turn. 

4.1.5 Controlling ||A|| 2 for unbounded classes 

For the remainder of the proof, we condition on the event A{X n ) n 7~(7 n ) H C{^ n ). We split our 
analysis into three cases. Throughout the proof, we make use of the quantity 

5 n : = B max(VsA n , (38) 

where B S (1, oo) is a constant to be chosen later in the argument. 
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Case 1: If ||A||2 < ||A|| n , then combined with inequality (37), we conclude that 
\\A\\ 2 n < 2ViA n ||A|| n + 24 v / iA n ||A|| n + I2s\ 2 n + 25sp n . 

This is a quadratic inequality in terms of the quantity ||A|| n , and some algebra shows that it implies 
the bound ||A|| n < 15max(y / iA n , ^/sp n ). By assumption, we then have ||A|| 2 < 15 max(y / sA n , ^/sp n ) 
as well, thereby completing the proof of Theorem 1. 

Case 2: If ||A||2 < 8 n , then together with the bound (37), we conclude that 

\\A\\l < 2VsA„,||A|| ri + 24y/sX n 5 n + l2sX 2 n + 25sp n . 

This inequality is again a quadratic in ||A|| n ; moreover, note that by definition (38) of 5 n , we have 
sA^ + sp n = 0(5%). Consequently, this inequality implies that ||A|| n < C5 n for some constant C. 
Our starting assumption implies that ||A|| 2 < S n , so that the claim of Theorem 1 follows in this 
case. 

Case 3: Otherwise, we may assume that ||A||2 > S n and ||A||2 > ||A|| n . In this case, the 
inequality (37) together with the bound ||A||2 > ||A|| n implies that 

\\A\\l < 2 v / iA n ||A|| 2 + 24 v / iA n ||A|| 2 + 12sA^ + 25sp n . (39) 

Our goal is to establish a lower bound on the left-hand-side — namely, the quantity ||A||^ — in terms 
of ||A||2. In order to do so, we consider the function class Q{X n ^Pn) defined by functions of the 
form g = Ylj=i 9ji an d such that 

A re ||3l|n,l + /9n||fl , |k,l ^ 4A n ||gs|| n) i + &pn\\gs\\H,l + ^ s Pn, (40a) 
||fl , s||i,Ti< 2Vs\\gsh + s\ n and (40b) 
HsIU < \\gh- (40c) 

Conditioned on the events A{^ n ), T("f n ) and C(j n ), and with our choices of regularization pa- 
rameter, we are guaranteed that the error function A satisfies all three of these constraints, and 
hence that A € G(X n ,Pn)- Consequently, it suffices to establish a lower bound on ||g|| n that holds 
uniformly over the class G(X n ,p n ). In particular, define the event 

B(X n ,p n ) :=\ \\g\\l > \\g\\l/2 for all g G G(K,Pn) where ||g|| 2 >5 n \. (41) 



The following lemma shows that this event holds with high probability. 

Lemma 3. Under the conditions of Theorem 1, there are universal constants Ci such that 

¥[B(X n , p n )\ > 1 - ci exp(-c 2 n 7 2). (42) 

We note that this lemma can be interpreted as guaranteeting a version of restricted strong 
convexity [27] for the least-squares loss function, suitably adapted to the non-parametric setting. 
Since we do not assume global boundedness, the proof of this lemma requires a novel technical 
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argument, one which combines a one-sided concentration bound for non- negative random variables 
(Theorem 3.5 in Chung and Lu [12]) with the Sudakov minoration [28] for Gaussian complexity. 
We refer the reader to Appendix D for all the details of the proof. 

Using Lemma 3 and conditioning B(X n ,p n ), we are guaranteed that ||A||^ > ||A|||/2, and hence, 
combined with our earlier bound (39), we conclude that 

II A||| < ViA n ||A|| 2 + 48 % /sA n || A|| 2 + 2As\ 2 n + 50sp n . 

Hence ||A|| n < ||A||2 < C max(y / sA n , y/sp n ), completing the proof of the claim in the third case. 

In summary, the entire proof is based on conditioning on the three events 7~(7n), -4(A n ) and 
B(X n ,p n ). Prom the bound (35) as well as Lemmas 1 and 3, we have 

P[T(7 n )nA{X n )nB(X n ,p n )nC{ ln )] > l - Cl exp ( - c 2ni l), 

thereby showing that max{||/ — /*||^, ||/ — /*|||} — C max(sA^, sp n ) with the claimed probability. 
This completes the proof of Theorem 1. 

4.2 Proof of Theorem 2 

We now turn to the proof of the minimax lower bounds stated in Theorem 2. For both parts (a) 
and (b), the first step is to follow a standard reduction to testing (e.g., [16, 38, 39]) so as to obtain 
a lower bound on the minimax error 9Jtp (.F^s/h) in terms of the probability of error in a multi-way 
hypothesis testing. We then apply different forms of the Fano inequality [38, 39] in order to lower 
bound the probability of error in this testing problem. Obtaining useful bounds requires a precise 
characterization of the metric entropy structure of J-d,s,Hi as stated in Lemma 4. 

4.2.1 Reduction to testing 

We begin with the reduction to a testing problem. Let {f 1 , . . . , f M } be a <5 n -packing of T in the ||-||2- 
norm, and let be a random variable uniformly distributed over the index set [M] : = {1, 2, . . . , M}. 
Note that we are using M as a shorthand for the packing number M (<5 n ; J- , \\ ■ ||2). A standard 
argument (e.g., [16, 38, 39]) then yields the lower bound 

inf sup P[||/ - r\\ 2 2 > 5 2 J2] > infP[G + 6], (43) 
/ e 

where the infimum on the right-hand side is taken over all estimators that are measurable 
functions of the data, and take values in the index set [M]. 

Note that P[0 7^ 0] corresponds to the error probability in a multi-way hypothesis test, where 
the probability is taken over the random choice of 0, the randomness of the design points X™ : = 
{xi}f =1 , and the randomness of the observations : = {yi}f=i- Our initial analysis is performed 
conditionally on the design points, so that the only remaining randomness in the observations 
comes from the observation noise {wi}" =1 . From Fano's inequality [13], for any estimator 0, we 

have P[0 7^ I X[ l J > 1 1 — t M , where Ixf(®',Y™) denotes the mutual information 
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between and Y™ with X" fixed. Taking expectations over X™, we obtain the lower bound 

E X n\l X n(B;Yi n )] +log2 

PB/8 >1- ' 1 J1 — . 44 

log M 

The remainder of the proof consists of constructing appropriate packing sets of J 7 , and obtaining 
good upper bounds on the mutual information term in the lower bound (44). 

4.2.2 Constructing appropriate packings 

We begin with results on packing numbers. Recall that log M(5; J- , \\ ■ || 2 ) denotes the 5-packing 
entropy of T in the || • H2 norm. 

Lemma 4. (a) For all 5 € (0, 1) and s < d/A, we have 

logM(6;T, || • || 2 ) = <D(s logM(^;B w (l), || • || 2 ) + slog-). (45) 

(b) For a Hilbert class with logarithmic metric entropy (20) and such that \\fW2 < 11/11%; there 
exists set {f 1 , . . . , f } with logM > C {slog(d/s) + sm}, and 

5 < ||/ fc -/ m || 2 < 85 for all k^ me {1,2,... ,M}. (46) 

The proof, provided in Appendix E, is combinatorial in nature. We now turn to the proofs of parts 
(a) and (b) of Theorem 2. 

4.2.3 Proof of Theorem 2(a) 

In order to prove this claim, it remains to exploit Lemma 4 in an appropriate way, and to upper 
bound the resulting mutual information. For the latter step, we make use of the generalized Fano 
approach (e.g., [39]). 

From Lemma 4, we can find a set {f\...J M } that is a 5-packing of J- in £ 2 -norm, and 

such that ||/ fc - / £ || 2 < 85 for all k,£ £ [M]. For k = 1,...,M, let Q k denote the conditional 

distribution of Y™ conditioned on X™ and the event {B = k}, and let D(Q k || (Q/) denote the 

Kullback-Leibler divergence. From the convexity of mutual information [13], we have the upper 

bound I X n(Q; Y™) < j^- Ylk£=i D(Q k \\ Q e ). Given our linear observation model (7), we have 
1 2 J 



D(<® k \\® e ) = ^ir(f\x l )-f( Xl )) 2 - n|l/fc f 

and hence 



2a 2 

i=l 



M M 

E x n[l x ?(Y?;e)] ^[\\f k - ffn] = ~ 2l k E \\f k -f\\l 

\ 2 ) k,l=\ \ 2 ) k,£=l 

Since our packing satisfies ||/ fc — / ||| < 645 2 , we conclude that 

Ex?[lx?(Yi',Q)] <32n<5 2 . 
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From the Fano bound (44), for any 5 > such that 32 "^ |g ^ 1 ° g 2 < ^, then we are guaranteed that 
P[9 7^ 0] > |. From Lemma 4(b), our packing set satisfies logM > C{sm + slog(d/s)}, so that 
so that the choice <5 2 = C {^p + iiH^M J ; f or a suitably small C > 0, can be used to guarantee 
the error bound P[9 / 9] > |. 

4.2.4 Proof of Theorem 2(b) 

In this case, we use an upper bounding technique due to Yang and Barron [38] in order to upper 
bound the mutual information. Although the argument is essentially the same, it does not fol- 
low verbatim from their claims — in particular, there are some slight differences due to our initial 
conditioning — so that we provide the details here. By definition of the mutual information, we have 

1 M 

k=l 

where Q fc denotes the conditional distribution of Y™ given 9 = k and still with Y™ fixed, whereas 
Py denotes the marginal distribution of Py . 

First we define covering numbers. Let (G,p) be a totally bounded metric space, consisting of a 
set Q and a metric p : Q x Q — > M + . An e-covering set of Q is a collection j/ 1 , . . . , f N } of functions 
such that for all / G Q there exists k G {1,2, N} such that p(f,f k ) < e. The e-covering number 
N(e; G, p) is the cardinality of the smallest e-covering set. 

Now let {g\...,g N } be an e-cover of J- in the II • ||o norm, for a tolerance e to be chosen. As 
argued in Yang and Barron [38] , we have 

M N 

/x P (9;Yn = _^Z)(Q^P y )<I)(Q*||-^), 
j=i k=i 

where P^ denotes the conditional distribution of Y™ given </ and X™. For each £, let us choose 
£*(k) G argmin^i^.^jy \\g^ — f k \\2- We then have the upper bound 

1 M 



M 

k= 



Taking expectations over Y™, we obtain 



E X n[i x „(&;Y?)} < j- J2 i lo s N + i E ^r Ols r(fc) - f k \\l]} 

k=l 

<logY + ^e 2 , 

where the final inequality follows from the choice of our covering set. 

From this point, we can follow the same steps as Yang and Barron [38]. The polynomial 
scaling (21) of the metric entropy guarantees that their conditions are satisfied, and we conclude 
that the minimax error is lower bounded any 5 n > such that n<5 2 > Clog N(5 n ; J-, \\ ■ [[2). From 
Lemma 4 and the assumed scaling (21), it is equivalent to solve the equation 



nbl > c{slog(d/ S ) + s(^/Sn, 



l/a 
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s lo (d/s') 1 

from which some algebra yields 5 2 n = C{^fZ£i + — } suitable choice. 
4.3 Proof of Theorem 3 

Recall the definition of s ^(-B) and %{S,B) from Section 3.5 which ensures /* is uniformly 
bounded by B. In order to establish upper bounds on the minimax rate in L 2 (P)-error over 
J~ds wC^)' we we anar y ze a least-squares estimator (not the same as the original M-estimator (10)) 
constrained to -n(B): 

n 

f G arg min V] (y t - y n - f{xi)) 2 . (47) 



i=l 



Since our goal is to upper bound the minimax rate in L 2 (P) error, it is sufficient to upper bound 
the L 2 (P)-norm of / — /* where / is any solution to (47). The proof shares many steps with the 
proof of Theorem 1. First, the same reasoning shows that the error A : = f — f* satisfies the basic 
inequality 

^ n 2 n 1 n 

-VA 2 (ii) < -| yVA(a?i)| + |i/ n -p||~y]A(si)|. (48) 
n L — 4 n L — ' n £ — 4 

i=l i=l i=l 

Recall the definition (25) of the critical rate 5 n . Once again, we first control the term error due 
to estimating the mean \y n — fi\. Noting that y n — \i = ^ Yl7=i(yi ~ M) an d the random variable 
yi — fj, = f*(xi) + wi is sub-Gaussian with parameter \JB + 1. This follows since /* is bounded 
by B and using standard results on sums of independent sub-Gaussian random variables (see e.g. 
Lemma 1.4 in Buldygin and Kozachenko [9]). Therefore 

nt 2 

n\Vn -fi\>t)< 2exp(- ). 



Setting A{5 n ) = {\y n — /•*!< VB5 n }, it is clear that 



PWy>l-2exp(-^). 



For the remainder of the proof, we condition on the event A(5 n ), in which case equation (29) 
simplifies to 

1 1 n 

-||A|| 2 < |- Y j w i A{x i )\ + y/BS n \\A\\ n . (49) 
i=i 

Here we have used the fact that 1^ Yn=i ^{ x i)\ < l|A||n, by the Cauchy-Schwartz inequality. 

Now we control the Gaussian complexity term |^ Y^i=i w i^( x i)\- For any fixed subset S, define 
the random variable 

1 ™ 

Z n {w,t;H(S,2B)) := sup I - V ty i A(x i ) I . (50) 

A£H(S,2B) n ~[ 
||A||„<t 

We first bound this random variable for a fixed subset S of size 2s, and then take the union bound 
over all ( 2 ) possible subsets. 
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Lemma 5. Assume that the RKHS H has eigenvalues (fJ>k)kLi that satisfy fik ~ k 2a and eigen- 
functions such that \\4>k\\oo < C. Then we have 



P[3t > such that Z n (w,t;U{S,2B)) > 16B C J 8 ^ bg S +3tS n ] < a exp(-9n^). (51) 

The proof of Lemma 5 is provided Appendix F.l. Returning to inequality (49), we note that by 
definition, 

2 n 



^WiA(xi)\ < max Z n (w,\\A\\ n ;H(S,2B)). 
■- i=1 \ s \- 2s 

Lemma 5 combined with the union bound implies that 



max Z n (w,\\A\\ n ;H(S,2B)) < 16B C \ / " bg S + 3S n \\ AIL 
\S\=2s V n 

with probability at least 1 — c\ („ ) exp(— 3n5^). Our choice (25) of 8 n ensures that this probability 
is at least 1 — c\ exp(— C2^(5^). Combined with the basic inequality (49), we conclude that 



< 32BC\ r 1/al ° gS +7B6 n \\A\\ n (52) 



with probability 1 — c\ exp(— c^nb^). 

By definition (25) of 5 n , the bound (52) implies that ||A|| n = 0(5 n ) with high probability. In 
order to translate this claim into a bound on ||A||2, we require the following result: 

Lemma 6. There exist universal constants (c, 01,02) such that for all t > c5 n , we have 

1 1 1 1 3 

! < HsIU < -H5II2 for all g en{S,2B) with \\g\\ 2 >t (53) 



2 " "• 7 "" " 2 
with probability at least 1 — c\ exp(— C2nt 2 ). 

Proof. The bound (53) follows by applying Lemma 7 in Appendix A with Q = H(S, 2B) and b = 2B. 

2 

The critical radius from equation (55) needs to satisfy the relation Q t0>n (e n ; TL{S, 2B)) < From 

Lemma 11, the choice e\ = 320 B C \J s 1 ra log -^ satisfies this relation. By definition (25) of 5 n , we 
have 5 n > ce n for some universal constant c, which completes the proof. □ 

This lemma implies that with probability at least 1— c\ exp(—C2Bnd 2 l ), we have || A||2 < 2||A|| n + 
C5 n . Combined with our earlier upper bound on ||A|L, this completes the proof of Theorem 3. 



5 Discussion 



In this paper, we have studied estimation in the class of sparse additive models defined by univariate 
reproducing kernel Hilbert spaces. In conjunction, Theorems 1 and 2 provide a precise characteri- 
zation of the minimax-optimal rates for estimating /* in the L 2 (P)-norm for various kernel classes 
with bounded univariate functions. These classes include finite-rank kernels (with logarithmic 
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metric entropy), as well as kernels with polynomially decaying eigenvalues (and hence polynomial 
metric entropy). In order to establish achievable rates, we analyzed a simple M-estimator based on 
regularizing the least-squares loss with two kinds of ^i-based norms, one defined by the univariate 
HUbert norm and the other by the univariate empirical norm. On the other hand, we obtained our 
lower bounds by a combination of approximation-theoretic and information-theoretic techniques. 

An important feature of our analysis is we assume only that each univariate function is bounded, 
but do not assume that the multivariate function class is bounded. As discussed in Section 3.5, 
imposing a global boundedness condition in the high-dimensional setting could lead to a substan- 
tially smaller function classes; for instance, for Sobolev classes and sparsity s = Cl(y/n), Theorem 3 
shows that it is possible to obtain much faster rates than the optimal rates for the class of sparse 
additive models with univariate functions bounded. Theorem 3 in our paper shows that the rates 
obtained under global boundedness conditions are not minimax optimal for Sobolev spaces in the 
regime s = Q(y/n). 

There are a number of ways in which this work could be extended. For instance, although 
our analysis was based on assuming independence of the covariates Xj, j = 1, 2, ... d, it would be 
interesting to investigate the case when the random variables are endowed with some correlation 
structure. One might expect some changes in the optimal rates, particularly if many of the variables 
are strongly dependent. This work considered only the function class consisting of sums of univariate 
functions; a natural extension would be to consider nested non-parametric classes formed of sums 
over hierarchies of subsets of variables. Analysis in this case would require dealing with dependencies 
between the different functions and is left for future research. 
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A A general result on equivalence of L 2 (P) and L 2 (P n ) norms 

Since it is required in a number of our proofs, we begin by stating and proving a general result that 
provides uniform control on the difference between the empirical || • || n and population || • ||2 norms 
over a uniformly bounded function class Q. We impose two conditions on this class: 

(a) it is uniformly bounded, meaning that there is some b > 1 such that ||g||oo < b for all g £ Q. 

(b) it is star-shaped, meaning that if g E G, then Xg £ G for all A £ [0, 1]. 

For each co-ordinate, the Hilbert ball B%(2) satisfies both of these conditions; we use Q = B%(2). 
(To be clear, we cannot apply this result to the multivariate function class J~ds~H-> since it is not 
uniformly bounded.) 

Let {<Tj}f =1 be an i.i.d. sequence of Rademacher variables, and let {xi}f =1 be an i.i.d. sequence 
of variables from X, drawn according to some distribution Q. For each t > 0, we define the local 
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Rademacher complexity 



1 - 

Qa,n{t,G) : = E Xi<7 [ sup -} j a i g(xi)] (54) 



Il!/ll2<* 71 i= l 

We let e n denote the smallest solution (of size at least 1 / y/n) to the inequality 

Qa,n(^,G) = ^, (55) 

where our scaling by the constant 40 is for later theoretical convenience. Such an c n exists, because 

the star-shaped property implies that the function Q a ,n{t, G)/t is non-increasing in t. This quantity 

corresponds to the critical rate associated with the population Rademacher complexity. For any 

t > e n , we define the event £(t) := {sup ge g \\\g\\ n — H5H2I > 

Nla<* 

Lemma 7. Suppose that \\g\\oo < b for all g € Q. Then there exist universal constants (01,02) such 
that for any t > e n , 

F[£ (t)] < ciexp(-c 2 nt 2 ). (56) 

In addition, for any g £ G with \\gW2 > i ? we /iaue ||<?|| n < IMl2(l + §)> an d moreover, for all g £ G 
with \\gW2 > W ; we /iaue 

1„ .3.. 

^Wgh < \\g\\n < ^\\g\\2, (57) 

both with probability at least 1 — c\ exp(— C2nt 2 ). 

Lemma 7 follows from a relatively straightforward adaptation of known results (e.g., Lemma 5.16 
in van de Geer [35] and Theorem 2.1 in Bartlett et al. [4]), so we omit the proof details here. 

B Proof of Lemma 1 

The proof of this lemma is based on peeling and weighting techniques from empirical theory [1, 
35] combined with results on the local Rademacher and Gaussian complexities [4, 25]. For each 
univariate Hilbert space %j = H, let us introduce the random variables 



1 n r 1 n 

Z n {w,t;%) : = sup |- ^ Wi9j(xjj)\, and Z n (w, t; H) : = E x sup \-y~]wigj(xjj] 
\\9j\\n<l n i=1 U\9j\\H<l n 4=1 

\\gA\n<t \\9jh<t 

(58) 

where Wi ~ N(0, 1) are i.i.d. standard normal. The empirical and population Gaussian complexities 
are given by 

Q w ,n(t,'H) : = E w [Z n (w;t,n)] and Q w , n {t,H) : = E w [Z n (w; t, %)] . (59) 

For future reference, we note that in the case of a univariate Hilbert space T~L with eigenvalues 
tVfcjfcLi) resu hs in Mendelson [25] imply that there are universal constants q < c u such that for 
all t 2 > 1/re, we have 

00 00 
01 [£min{t 2 ,,x fc }] 1/2 < Q w , n (t,n) < -^[J>in{i 2 ,^}] 1/2 , (60) 



k=l V k=l 
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for all j The same bounds also hold for the local Rademacher complexities for Reproducing kernel 
Hilbert spaces. 

Let P n j > denote the smallest positive solution r of the inequality 

Qw,n(r,-H) <4r 2 . (61) 

The function Q Wt n( r ,'H) defines the local Gaussian complexity of the kernel class in co-ordinate 
j. Recall the bounds (60) that apply to both the empirical and population Gaussian complexities. 
Recall that the critical univariate rate v n is defined in terms of the population Gaussian complexity 
(see equation (13)). 



B.l Some auxiliary results 

In order to prove Lemma 1, we also need some auxiliary results, stated below as Lemmas 8 and 9. 
Lemma 8. For any function class Q and all 6 > 0, we have 

n5 2 

¥[\Z n (w,t,G) - Q w , n (t,g)\ >St] <2exp(-— ), and (62a) 
F[\Z n (w,t,g) - Q w , n {t,G)\ > St) < 2expf - — J. (62b) 

Proof. We have 

- 1 n t 

\Z n (w,t,G) - Z n (w',t,g)\ < sup -| }(wi -v/ i )g(x i )\ < —=\\w-w' 



h\U<t 



2, 



showing that Z n (w,t,g) is -^-Lipschitz with respect to the l 2 norm. Consequently, concentration 
for Lipschitz functions of Gaussian random variables [21] yields the tail bound (62a). Turning to 
the quantity Z n (w,t,H), a similar argument yields that 



1 n 

\Z n (w,t,g) - Z n (w',t,g)\ < E x [ sup -IVY™* -w' i )g{x i 

hh<t l ~ L 
1 n 

< sup E x [(- V<7 2 (^)) V2 ] \\w-w'\\ 2 < -= 



\\9h<t 



\w - w'\\ 2 , 



where the final step uses Jensen's inequality and the fact that E x [(7 2 (xj)] < t 2 for all i = 1, . . . ,n. 
The same reasoning then yields the tail bound (62b). □ 

Our second lemma involves the event T>(pi n ) : = {v n .j < 7n> for all j = 1,2, ... where we 



recall the definition (61) of v n j, and that j n : = ftmax [u n , y ^p}- 
Lemma 9. For all 1 < j < d, we have 

^\pn,j < In] > 1 - ci exp(-c 2 n7 2 ). (63) 
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Proof. We first bound the probability of the event {y n ,j > 7n} f° r a fixed T~Lj. Let g £ . (1) be 
any function such that \\gW2 > t > v n . Then conditioned on the sandwich relation (57) with 6 = 1, 
we are guaranteed that \\g\\ n > §• Taking the contrapositive, we conclude that ||g|| n < | implies 
II9II2 < t, and hence that Z n (w,t/2,H) < Z n (w,t,'H) for all t > u n , under the stated conditioning. 

For any t > v n , the inequalities (57), (62a) and (62b) hold with probability at least 1 — c\ exp(— C2nt 2 ). 
Conditioning on these inequalities, we can set t = "f n > u n , and thereby obtain 

(a) ~ 

Qw,n{ln,'H) < Z n (w,J n ,H) + ■jl 
(b) 

< Z n (w, 2j n , H) +7 n 

(c) 

id) „ 
< 

where inequality (a) follows from the bound (62a), inequality (b) follows the initial argument, 
inequality (c) follows from the bound (62b), and inequality (d) follows since 2~/ n > e n and the 
definition of e n . 

By the definition of v n j as the minimal t such that Q w ,n{t,T~L) < At 2 , we conclude that for each 
fixed j = l,...,n, we have u n j < 7 n with probability at least 1 — c\ exp(— 02^,7^). Finally, the 

uniformity over j = 1, 2, . . . , d follows from the union bound and our choice of 7„ > k^/^^. 

□ 



B.2 Main argument to prove Lemma 1 

We can now proceed with the proof of Lemma 1. Combining Lemma 9 with the union bound over 
j = 1,2, ... ,d, we conclude that that 

Pp?(7n)] > l-c 1 exp(- C2 n 7 2), 

as long as c% > 1. For the remainder of our proofs, we condition on the event T){^ n ). In particular, 
our goal is to prove that 

1 - 

|~2>*£M < C {jl \\fj\\n + 7n WfjWn} for all fj G H (64) 

i=l 

with probability greater than 1 — c\ exp(— 02^17^). By combining this result with our choice of 7 n 
and the union bound, the claimed bound then follows on P[T(7 n )]- 

If fj = 0, then the claim (64) is trivial. Otherwise we renormalize fj by defining gj : = fj/\\fj\\Hi 
and we write 

^ n \ n 

~^2 w ifj( x ij) = WfjWn -*52 w i9j(xij) < WfjWn Z n (w;\\g j \\ n ,'H), 
n i=i n i=i 

where the final inequality uses the definition (58), and the fact that = 1. We now split the 

analysis into two cases: (1) ||<7j|| n < 7n> an d (2) \\gj\\ n > In- 
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Case 1: \\gj\\ n < In- In this case, it suffices to upper bound the quantity Z n (w,^f n ,'H). Note 
that = 1 and recall definition (58) of the random variable Z n . On one hand, since 7„ > u n j 

by Lemma 9, the definition of u n j implies that Q w ,n{^n,T~i) < ^ln-> anci hence 

E[Z n (w; ln ;U)] = Q w , n ^n,n) < 4jI 

Applying the bound (62a) from Lemma 8 with 5 = 7 n = t, we conclude that Z n (w;j n ;T-L) < 
C 7^ with probability at least 1 — c\ exp { — 02/17^}, which completes the proof in the case where 

Iblln < In- 

Case 2: \\gj\\ n > 7«. In this case, we study the random variable Z n (w;rj-,'H) for some rj > j n . 
Our intermediate goal is to prove the bound 



Z n {w\r^U) > Crjj v 



< ci exp { - c 2 «7^}. (65) 



Applying the bound (62a) with t = rj and 5 = j n , we are guaranteed an upper bound of the 
form Z n (w; rj;7i) < Qw,n(j'j,'H) + rj j n with probability at least 1 — c\ exp ( — C2n7^). In order to 
complete the proof, we need to show that Q w <n (r j- ,~H) < rj j n . Since rj > j n > v n ,ji we have 



n 

Tn „ r 1 1 



Qw,n{rj,H) = t^-^w[ sup |- y^ y Wj9j(xij)\) < ^ J -Q w ,n(vn,j,'H) < 4rjV nj j, 

Un >3 \\9j\\n<Vn,j 71 j = i U n,3 

llsjllw^^ 2 - 



where the final inequality uses the fact that Q w ,n(pn,j,'H) < 4z^ •. On the event T>{^ n ) from 
Lemma 9, we have u n j < 7 n , from which the claim (65) follows. 

We now use the bound (65) to prove the bound (64), in particular via a "peeling" operation 
over all choices of 77 = WfjWn/WfjWu- (See van de Geer [35] for more details on these peeling 
arguments.) We claim that it suffices to consider 77 < 1. It is equivalent to show that \\gj\\ n < 1 
for any gj E M H (l). Since \\gj\\oo < hjllu < 1, we have \\gj\\ n = ~ Th=i d) ( x ij) < 1> as required. 
Now define the event 

W:=k-6B«(1) I £f>/;M > 8 ||/;||« 7n j^K and^^G( 7 „,l]). 

I \\jj\\n \\Jj\\H J 

and the sets S m : = {2 m - 1 -y n < < 2 m 7n } for m = 1, 2, . . . , M. By choosing M = 2 log 2 (l/7„), 

we ensure that 2 M j n > 1, and hence that if the event Tj{^ n ) occurs, then it must occur for function 

II f II 

fj belonging to some S m , so that we have a function /,• such that y " < t m : = 2 m -f n , and 

J J WljW'H 

\-Y,W i f j { Xij )\> 8 ||/,|| W7 n^ i >C\\fj\\ U t m , 

which implies that Z n (w\t m ,'K) > 4i m . Consequently, by union bound and the tail bound (65), we 
have 

p [75'(7n)] < M ci exp { - c 2 n7^} < ci exp { - c' 2 ?i^} 
by the condition 717^ = 0(log(l/7„)), which completes the proof. 
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C Proof of Lemma 2 



Define the function 

£(A) : = -L (Wi + /i + y n ~ A(x 4 )) 2 + A„||r + A|| n ,i + pn\\f* + A||«,i 



2n 



and note that by definition of our M-estimator, the error function A : = / — /* minimizes C From 



the inequality £(A) < £(0), we obtain the upper bound i||A||* <Ti+ T 2 , where 



^ n 1 n 

Ti : = \-} j w i A(x i ) \ + \y n - /u||- A(xi)|, and 



n i — ' n 



d d 

T 2 -.=\ n Y,{\\fj\\n - ii/; + + - ii/; + 4-im- 
j=i j=i 

Conditioned on the event C{^ n ), we have the bound \y n — A*||^ X)"=i A(#i)| ^ v / *7™||A|| n , and hence 
|||A||£ < T 2 + |i X7=i WiA(xj)| + v / s7 n ||A|| n , or equivalently 

1 1 n 1 

< -(||A|| ri -^7n) 2 <T 2 + |-E^A( 2;i )| + - S 7n- (66) 

i=i 

It remains to control the term T 2 . On one hand, for any j £ S c , we have 

||/;iU-||/; + ^-||„ = -IIA.IU, and H/JI^-II/J + A,-^ = 

On the other hand, for any j E S, the triangle inequality yields H/^Hn — \\fj + Aj|| n < ||Aj|| n , with 
a similar inequality for the terms involving || • ||%. Combined with the bound (66), we conclude 
that 



1 ~ 1 

< -Y,wA(xi) + A n {||A 5 |U,i - ||A 5 c|Ui} + Pn {\\A s \\ Htl - \\A s 4h,i} + 2 S 7n- (67) 

i=l 

Recalling our conditioning on the event T(7n), by Lemma 1, we have the upper bound 

1 n 

|- 5>iA(*i)| < 8 {7n||A|| n ,i + 7nl|A|ki}- 

i=l 

Combining with the inequality (67) yields 

< 8 {7n||A|| n ,i +7nl|A|kl} + An{||A s || n ,i - ||A S c|| n!l } +p n {\\K s \\ n>1 - ||A 5 c||«,l} + ^7n 

< ^||A|| n!l + ^IIAH^ + A4||A 5 |U,i - ||A 5c || n!l } + p n {WK s \\u,i - ||^l|«,i} + \ s ^l 
where we have recalled our choices of (X n ,p n ). Finally, re-arranging terms yields the claim (33). 
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D Proof of Lemma 3 



Recalling the definition (40) of the function class G(X n ,p n ) and the critical radius 5 n from equa- 
tion (38), we define the function class G'(X n , p n , 5 n ) := {/i 6 G(X n ,p n ) \ \\h\\2 = 5 n }, and the alter- 
native event 

B'(X n ,p n ):={{\\h\\ 2 n > 5 2 n /2 for all h G G'(X n , p n , £„)}■ 

We claim that it suffices to show that B'(X n , p ri ) holds with probability at least 1 — c\ exp(— C2/17 2 ). 
Indeed, given an arbitrary non-zero function g G G(X n , p n ), consider the rescaled function h = j^^9- 

Since g G G(X n ,p n ) and G(X n ,p n ) is star-shaped, we have h G G(X n ,p n ), and also \\h\\2 = S n by 
construction. Consequently, when the event B'{X n ,p n ) holds, we have ||/i||^ > <%/2, or equivalently 
Iblln — Iblli/^) showing that B(X n ,p n ) holds. Accordingly, the remainder of the proof is devoted 
to showing that B'(X n ,p n ) holds with probability greater than 1 — c\ exp(— C2W-f 2 ). Alternatively, 
if we define the random variable Z n (G') '■ = sup^g, {<5 2 — ^ 2~27=i f 2 ( x i)}i then it suffices to show 
that Z n (G') < 1 2 J2 with high probability. 

Recall from Section 4.2.4 the definition of a covering set; here we use the notion of a proper 
covering, which restricts the covering to use only members of the set G- Letting N pr (e; G, p) denote 
the propert covering number, it can be shown that N pi (e;G,p) < N(e;G,p) < N pr (e/2;G,p)- Now 
let g 1 , . . . ,g N be a minimal 5 n /8-proper covering of G 1 in the L 2 (P n )-norm, so that for all / G G' , 
there exists g = g k G G' such that ||/ — g\ n < 5 n /8. We can then write 



-I / * -1 1 1 -. it 

A*) = {% - 1 E^)} + {- E ofo*) - f 2 (-m- 

n L — ' n L — ' n L — ' 

i=l i=l i=l 

By the Cauchy-Schwartz inequality, we have 

-. n 1 n 

- £ (g 2 &) - f 2 ( x *)) = - E (ate) - f(?i)Mxi) + f(*i)) 



i=l 



n 



i=l 



< 



1 n 



/C*)) s 



i=i 



1 n 

-Y(f(x l )+g(x l W 



i=l 



Is -/II. 



n 

-£(/(x 4 )+5(^)) 2 - 



i=l 



By our choice of the covering, we have \\g — f\\ n < 5 n /8. On the other hand, we have 



1 n , / 

-Y J {f{x l ) + g{x l )Y<^/2\\f\\l + 2\\g\\l < ^4P n 



25„ 



i=l 



where the final inequality follows since 



\g\\n = 5 n . Overall, we have established the upper 



bound - Yli=i (d ( x i) ~ f ( x i)) — "f 1 ) an d hence shown that 



1 n I 2 
Z n {G')< i max w {^--E^))} + f, 
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where N = N pi (5 n /8, Q' , \\ ■ || n ). For any g in our covering set, since g 2 (xi) > 0, we may apply 
Theorem 3.5 from Chung and Lu [12] with t = <5^/4 to obtain the one-sided tail bound 

j-^Aisfi^i-^ («) 

where we used the upper bound vav(g 2 (x)) < E[g 4 (x)]. Next using the fact that the variables 
{gj{xj)}j =1 are independent and zero- mean, we have 

E[gHx)] = ^E^M+^^E^CxyJMCxfc)] 

j=l V 7 &k 

< 4^EbJ(^)] + 6X;E[ 5 |(^)]E E ^(^)] 

J'=l 3=1 fc=l 

< 4^ + 65* 

< 



J n 1 ""n 
'n ' 



where the second inequality follows since ||<7j||oo — WdjWy. — 2 for each j. Combining this upper 
bound on E[<7 4 (x)] with the earlier tail bound (68) and applying union bound yields 



1 n X% 

max )jv {5l - -Y.ftzt)} > -f] < exp (logiV pr (^/8, Q' , \\ ■ \\ n ) - ^g). (69) 

i=l 



It remains to bound the covering entropy log N pi (5 n /8, Q' , \\ ■ \\ n ). Since the proper covering 
entropy log N pT (5 n /8, Q' , \\ ■ \\ n ) is at most logiV(J n /16, Q', \\ • || n ), it suffices to upper bound the 
usual covering entropy. Viewing the samples (x±, X2, x n ) as fixed, let us define the zero-mean 
Gaussian process {W g ,g € Q'} via 

w 9 ■ = 7^ E?=i eigfa), where the variables {£i}™ =1 are i.i.d. 
standard Gaussian variates. By construction, we have var[(W ff — W/))] = \\g — /||^. Consequently, 
by the Sudakov minoration [28], for all e > 0, we have eydog iV(e; Q', \\ ■ \\ n ) < 4E e [sup 9G g/ W g \. 
Setting e = 8 n /lQ and performing some algebra, we obtain the upper bound 

1 / 64 1 

— JlogiV(£ n /16;£',|| • lln) < ~-E £ [sup- V^^)]. (70) 



i n 



i=l 



The final step is to upper bound the Gaussian complexity E £ [sup ^ Y17=i £ i9( x i)]- I n the proof of 

g&S' 

Lemma 1, we showed that for any co-ordinate j £ {1,2, ... ,d}, the univariate Gaussian complexity 
is upper bounded as 



1 n 

E[ sup -^2si9j(xij)] < C {jnrj +^Rj}. 



\\9j\\n<rj n i—l 

hj\\n<Rj 
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Summing across co-ordinates and recalling the fact that the constant C may change from line to 
line, we obtain the upper bound 



E £ [sup - ^r,£ig(xi)] < Csup {7„ \\g\\i, n + ll\\g\\i,u} 
geQ' n ~[ geg' 

0) 1 

< C SUp {4j n \\gs\\l,n + 4ln\\gs\\l,H + l^SPn} 

geG' 61 

(b) 

< C SUp {7„ Hpslll.n + Sp n ) 

g&G' 

w r 

< C SUp< J n [2Vs|bl|2 + S-f n ] + Sp r 

geG' I 

where step (a) uses inequality (40a) in the definition of Q'\ step (b) uses the inequality ||c7j||-h < 2 
for each co-ordinate and hence ||<?s||i/h < 2s, and our choice of regularization parameter p n > 7^; 
and step (c) uses inequality (40b) in the definition of Q' . Since \\gW2 = 5n for all g € Q' , we have 
shown that 

1 .A ( d ) ^ 2 P 

E e [ S up-y2e t g(xi)]<C{ S1 2 n + ^Jn5n + sp n } < C{-^ + ^}, (71) 

where inequality (d) follows from our choice (38) of 5 n , and the constant B can be chosen as large 
as we please. In particular, by choosing B sufficiently large, and combining the bound (71) with 
the Sudakov bound (70), we can ensure that 

- iogiv(Vi6 ; g',||-y <-§-. 

n 640 



Combined with the earlier tail bound (69), we conclude that 

^2 

g\xi)} > - 

which completes the proof of Lemma 3. 



^«-;;iyM}S?lS«p(-{§) 



E Proof of Lemma 4 

Proof of part (a): Let N = M(4-; %(1), || • || 2 ) - 1, and define 1 = {0,1,..., N}. Using 
\\u\\o = Ylj=i^[ u j 7^ 0] to denote the number of non-zero components in a vector, consider the set 

&: = {uel d I ||u|| = s}. (72) 

Note that this set has cardinality \&\ = (^jN s , since any element is defined by first choosing s 
co-ordinates are non-zero, and then for each co-ordinate, choosing non-zero entry from a total of 
./V possible symbols. 
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For each j = l,...,d, let {0, fj, /?,..., /• } be a ^/"v/s-packing of B-^(l). Based on these 
packings of the univariate function classes, we can use 6 to index a collection of functions contained 
inside J 7 . In particular, any a£6 uniquely defines a function g u = Ylj=i 9j' ^ J- ■> with elements 

tfJff if "^° (73) 
10 otherwise. 

Since ||it||o = s, we are guaranteed that at most s co-ordinates of g are non-zero, so that g G J 7 . 

Now consider two functions 5" and h v contained within the class {g u ,u G ©}. By definition, 
we have 

\\9 U -h v \\l = J2\\f?-f?\\l > 7X>^], (74) 
i=i i=i 

Consequently, it suffices to establish the existence of a "large" subset i C 6 such that the 
Hamming metric pn{u, v) : = Sj=i ^[ u j v j] i s a * least s/2 for all pairs u,»ei, in which case we 
are guaranteed that \\g — > 5 2 . For any it G (3, we observe that 



{u G 6 I Ph(u,v) < -} 



This bound follows because we simply need to choose a subset of size s/2 where u and v agree, and 
the remaining s/2 co-ordinates can be chosen arbitrarily in (N + l) 5 ways. For a given set A, we 
write ph(u, A) < § if there exists some v G A such that ph(u, v) < | . Using this notation, we have 



{«G6 I ph(u,^4) < ^} 



<N (f) + <|6|> 



where inequality (a) follows as long as 

. 1 G) N° 



\A\ < N* 



2(f) (N + iy/ 2 ' 



Thus, as long as \A\ < N*, there must exist some element u G & such that pn(u,A) > |, in which 
case we can form the augmented set A U {u}. Iterating this procedure, we can form a set with TV* 
elements such that ph(u,v) > | for all u, v G A. 
Finally, we lower bound N* . We have 

N > - I ) 2 — - — 

- 2 1 s/2 ' (N + l) s / 2 

2 1 s/2 ' l iV + l J 

1 ,d — S N £ „ TQ /o 

- 2 y s/2 ' 

where inequality (i) follows by elementary combinatorics (see Lemma 5 in the paper [30] for details). 
We conclude that for s < d/A, we have 

d S 
logiV* =n( 8 bg- + *logM(-=;B«(l),|| • || 2 )), 
s ys 

thereby completing the proof of Lemma 4(a). 



32 



Proof of part (b): In order to prove part (b), we instead let N = M(|;B^(1), || • H2) — 1, and 
then follow the same steps. Since log N = fi(m), we have the modified lower bound 

log N* = U (s log — h sm) , 

s 

Moreover, instead of the lower bound (74), we have 

\\g» -h'\\l = Ell/?'-/?' Ill > li^^vj] > * (75) 

3=1 3=1 

using our previous result on the Hamming separation. Furthermore, since \\fjW2 < ll/jllw f° r an y 
univariate function, we have the upper bound 

lis" -^1! = Ell/?-/? Hi ^ En/? -/?!!«• 

3=1 3=1 

By the definition (72) of &, at most 2s of the terms /" J — fj j can be non-zero. Moreover, by 
construction we have \\fj j — /?!!% < 2, and hence 

\\g u -h v \\l<8s. 

Finally, by rescaling the functions by \^8 5/^, we obtain a class of N* rescaled functions {g u , u £ Z} 
such that 



Iflf-^Hl >5 2 , and Hp" -h v \\l < 64<5 2 , 



as claimed. 



F Results for proof of Theorem 3 

The reader should recall from Section 3.5 the definitions of the function classes J~d s u(B) and 
H(S,B). The function class %(£,£>) can be parameterized by the two-dimensional sequence 
{aj^k)j£S ,fceN of co-efficients, and expressed in terms of two-dimesnional sequence of basis func- 
tions {4>j^)j^s ,fceN an d the sequence of eigenvalues (nk)k<=N for the univariate RKHS H as follows: 

00 00 ffi 2 

U{S,B) :={f = EE a i>^ I E— ^ 1 V 3 e ^ and ||/||oo < 

jGSfc=l k=l ^ k 

For any integer M > 1, we also consider the truncated function class 

M 00 a 2 

-H(S,B,M):={f = EE a i.^> I E — < 1 Vj G 5and ||/||oo < B}. 

j£S k=l k=l ^ k 

Lemma 10. We have the inclusion %{S,B,M) C {/ e ^(5) | EjesEfcli K.fcl < B\/M}- 
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Proof. Without loss of generality, let us assume that S = {1,2, s}, and consider a function 
/ = Y2j=ifj ^ %{S,B,M). Since each fj acts on a different co-ordinate, we are guaranteed that 
ll/lloo = X^=i 1 1 /j I loo- Consider any univariate function fj = J2k=i a j,k<Pj,k- We have 

M _ / M \ V2 (a) 

k=l ^ k=l ' 

where step (a) uses the fact that E[fj(Xj)] = Y^k=i a2 jk — Yl!k=i a2 jk f° r an y M > 1. Adding up 
the bounds over all co-ordinates, we obtain 

s M s 

IHIi = 22 M ^ >^2ll/;IU = ^ll/lloo < Vmb, 

i=l k=l j=l 

where the final step uses the uniform boundedness condition. □ 



F.l Proof of Lemma 5 

Recalling the definition of Z n {w;t, / H{S ,2B)) stated from (50), let us view it as a function of the 
standard Gaussian random vector {w\, . . . ,w n ). It is straightforward to verify that this variable is 
Lipschitz (with respect to the Euclidean norm) with parameter at most t/y/n. Consequently, by 
concentration for Lipschitz functions [21], we have 

- ~ 9n5 2 

¥[Z n {w;t,n(S,2B)) > E[Z n (w; t,H(S, 2J3))] + 3t8 n ) < exp ( - -^). 

Next we prove an upper bound on the expectations 

1 n 

Qw,n(t;'H(S, 2B)) := E m [ sup -y^Wig{xi)], and (76a) 

gen(S,2B) n i=1 

Il9l|n<* 

1 " 

Q Win (t;H(S,2B)) := E x>w [ sup -VVste)]. (76b) 

g&H(S,2B) n 

\\gh<t 

Lemma 11. Under the conditions of Theorem 3, we have 



raax{Q w>n (t;H(S,2B)), Q w>n (t;H(S,2B))} < SBC 

Proof. By definition, any function g £ 7i(S,2B) has support at most 2s, and without loss of 
generality (re-indexing as necessary), we assume that S = {1, 2, 2s}. We can thus view functions 
in H{S, 2B) as having domain M? s , and we can an operator <3? that maps from M 2s to [^ 2 (N)] 2s , via 

x i y ^jf c (x) = 4>j^{xj), for j = 1, ... , 2s, and feel. 

Any function in g G H(S, 2B) can be expressed in terms of two-dimensional sequence (a^) and the 
functions ($j )fc ) as g{x) = g(x 1 ,x 2 , ■ . .,x 2s ) = J2T=i $j,k(x)a jtk = a)), where ((•, •)) 
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is a convenient shorthand for the inner product between the two arrays. 



For any function g £ ~H(S, 2B), triangle inequality yields the upper bound 

1 n 1 n 

SUp -\> Wi(($(xi),a}}\< SUp -| y^ Wi(($ l:M(%i), 0.,l:Af}} +M (77) 



g£2H(S,2B) n 



i=l 



g&2H(S,2B) n . =± 



where A 2 ■ = sup g&2n{S2B) -| E^ =1 ttfi((*.,M+i:oo(!Ct), a.,Af+l:oo»|- 

Bounding term E I)lu [Ai] and E„,L4i]: By Holder's inequality and Lemma 10, we have 



Ai<—= sup a..i;M 1,1 max| } —9 jk { 

V n g&2H(S,2B) ' h k ~ V n 



2 B VM 

Xi)\ < 1= — max | 

n j,k 



i=l 



x»)|- 



By assumption, we have |$j,fc(xj)| < C for all indices (i,j,k), implying that Y17=i ^^j,k( x * s 
zero-mean with sub-Gaussian parameter bounded by C and we are taking the maximum of 2s x M 
such terms. Consequently, we conclude that 



E w [Ai] < 8BC 



Mlog(Ms) 



(78) 



The same bound holds for E^f-Ai] 



Bounding term E^f^] and E^f^] 1 In order to control this term, we simply recognize that 
it corresponds to the usual Gaussian complexity of the sum of 2s univariate Hilbert spaces, each of 
which is an RKHS truncated to the eigenfunctions {nk}k>M+i- 



j n ^ 2s n 

-\^2wi((<&;M+l:oo(xi), a.,M+l:oo))| < ~7=^\ E a 3,k®j,k(%)^2 
n i=l ^ n j =1 k>M+l i=l 

2s 



< 



c 



Ei E 

j=l k>M+l i=l 



Wi_ 



a lk 



E *<E£ 

k>M+l i=l 



n 



where the final inequality follows from Cauchy- Schwartz. Exploiting the fact that X^fe>M+i — 1 
for all j, we have the bound 



One again a similar bound holds for E x ^[^2]- 



Z^fe>M+l Mfc 



(79) 
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Substituting the bound (78) and (79) into the upper bound (77), we conclude that 



Q w , n (m(S,2B)) < ABC X l Ml0giMs) + 4C S] /^+^* 



n V n 



n V n 

-2a 



where the second inequality follows from the relation ~ k~ . Finally, setting M = s« yields 
the claim. 

Note that the same argument works for the Rademacher complexity, since we only exploited 
the sub-Gaussianity of the variables lOj. □ 

Returning to the proof of Lemma 5, combining Lemma 11 with the bound (62a) in Lemma 8: 



[Z n (w;t,H(S,2B)) >8BC\ — + 3t5 n ] < exp ( - — 



Since < 2B for any function g 6 %{S,2B), the proof Lemma 5 is completed using a peeling 
argument over the radius, analogous to the proof of Lemma 1 (see Appendix B). 
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